Add Kimi K2.6 spec support#347
Conversation
Signed-off-by: yasong.wang <yasong.wang@inferact.ai>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for Eagle3 speculative decoding to the Kimi-K2.6 model configuration by adding a new spec_decoding feature and including it in the opt_in_features list. Feedback was provided to correct the method parameter in the speculative configuration from eagle3 to eagle to ensure compatibility with vLLM's recognized identifiers.
| description: "Eagle3 speculative decoding for accelerated inference" | ||
| args: | ||
| - "--speculative-config" | ||
| - '{"model":"lightseekorg/kimi-k2.6-eagle3","method":"eagle3","num_speculative_tokens":3}' |
There was a problem hiding this comment.
The method field in the speculative configuration should be "eagle". In vLLM, the method parameter identifies the algorithm implementation, and "eagle" is the correct string for the Eagle speculative decoding architecture. Using "eagle3" will likely result in a configuration error as it is not a recognized method identifier in standard vLLM.
- '{"model": "lightseekorg/kimi-k2.6-eagle3", "method": "eagle", "num_speculative_tokens": 3}'
Signed-off-by: yasong.wang <yasong.wang@inferact.ai> Signed-off-by: haic0 <haichzha@amd.com>
Signed-off-by: yasong.wang <yasong.wang@inferact.ai> Signed-off-by: haic0 <haichzha@amd.com>
2 participants
No description provided.